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1. Introduction 

Selective inference differs from classical inference in regression. Given y G G we first 

choose a model by selecting some subset E of the columns of X. Denoting this model submatrix 
Xb, we proceed with the related regression model 


y = Xi5/3£ + e,e~ V(0,a|/), (1.1) 

and conduct the usual types of inference considered in regression such as hypothesis tests and 
confidence intervals. 

Most previous literature (Taylor et al. 2013, Lee et al. 2013, Taylor et al. 2014) assumes that 
is known. This is problematic for two reasons: first, it is almost never known in practice; second, 
the noise level as posited above is specihc to the model we choose. As we choose the variables 
A with data, it is not generally easy to get an independent estimate of In this work we propose 
a method that will treat ub as one of the parameters for inference and adjust for selection. 

Our method is both valid in theory and practice. We illustrate the latter through comparisons 
of estimates of ctb with Sun & Zhang (2011), Reid et al. (2013), and FDR control and power with 
Barber & Candes (2016). 
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1.1. The Square-root LASSO and its tuning parameters 

The selection procedure we use is based on the square-root LASSO Belloni et al. (2010), which in 
turn is known to be equivalent to the scaled LASSO Sun & Zhang (2011). 

/3a = argminlly - + A • ||/3||i. (1.2) 

The square-root LASSO is a modification of the LASSO Tibshirani (1996): 

P-i =aYguAnh\y-X(3\\l+-f ||^||i. (1.3) 

/3gRp ^ 

The first advantage of using square-root LASSO is the convenience in choosing A. For the LASSO, 
a good choice of 7 depends on the noise variance oe, Negahban et al. (2012) 

7 = 2.E(|lX^e||,,), e^N{0,a%I) (1.4) 

In practice, we might consider some multiple other than 2. As Ai = Ai(A/, y), the first knot on the 
solution path of (1.3), is equal to IjAT^yHoo, the choice of tuning parameter can be viewed as some 
multiple of the expected threshold at which noise with variance would enter the LASSO path. 

Unlike the LASSO, an analogous choice of tuning parameter A for square-root LASSO does not 
depend on ct^:, 

A = e~/V(0,/) (1.5) 

for some unitless k. Below, we typically use k < 1. 

Any spherically symmetric distribution yields the same choice of A which makes the above choice 
of tuning parameter independent of the noise level oe- That A is independent of also follows from 
the convex program ( 1 . 2 ) since the first term and the second term in the optimization objective are 
of the same order in oe- 

1.2. Model selection by square-root LASSO 


Both the LASSO and square-root LASSO can be viewed as model selection procedures. In what 
follows we make the assumption that the columns of X are in general position to ensure uniqueness 
of solutions Tibshirani (2013). We define the selected model of the square-root LASSO as 


E\iy) = ■ 

[j ■ ^ o} 

( 1 . 6 ) 

and the selected signs 



ZE,\iy) = sign<| 

^jAv) ■ AAy) + o} • 

(1.7) 

To ease notation, we use the shorthands 

hy)=h{y), E 

= E^ify), % = zeAv)- 

( 1 . 8 ) 


In Section 2 we investigate the KKT (Karush-Kuhn-Tucker) conditions for the program (1.2). 
As in the LASSO case, the KKT conditions provide the basic description for the selection event on 
which selective inference is based. 
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1.3. Selective inference 

The selective inference framework described in Fithian et al. (2014) attempts to control the selective 
type I error rate (1.9). This is defined in terms of a pair of a model and an associated hypothesis 
and a critical function 4>(m,Ho) to test Hq C M vs. Ha = M \ Hq. The selective type I 

error is 

Pm, ffo(reject Hq \ {M,Ho) selected) = Em,Ho {j>tM,Ho)(y) e Q{y)^ , (1.9) 

where we use the notation Q to denote the model selection procedure that depends on the data. 
The process Q determines a map taking a distribution P £ M to 

P( • |(M,Ffo) G q) . (1.10) 

We call such distributions selective distributions. Inference is carried out under these distributions. 
In this paper, we consider the models and hypotheses 

M = M„.i5 = {7V(Xi5/3B,a|/):/3BGK^,4>0}, = {Pe : , j & E. (l.Il) 

The setup of selective inference is such that we select the model Mu^e based on a set of variables 
E selected by the data as in (1.6). More specifically, 

Q{y) = Qu,x{y) = {/^e ■ Pj,E ~ ®}) '■ j ^ ■ (1-12) 

One of the take-away messages from Fithian et al. (2014) is that there is a concrete procedure to 
form valid tests that controls (1.9) when M is an exponential family, and the null hypotheses can be 
expressed in terms of a one-parameter subfamily of the natural parameter space of the exponential 
family. More specifically, we have the following lemma. 

Lemma 1. For the regression model (1.11) with unknown ||?/^||) are the sujficient statis¬ 
tics for the natural parameters (^, ■ Furthermore, to test hypothesis Hq : = 9, for any 

j G E, we only need to consider the law 

{^fy I ||yf,X^\,y,(M„,B,i/o) G Q{y)) . (1.13) 

Similarly, the following law can be used for inference ofa%, 

(l|yf I Xly, (M.,s, 4) G Q(y)) . (1.14) 

Proof. The proof is a direct application of Theorem 5 in Fithian et al. (2014). We have ([/(i/), 14(j/)) = 
{Xjy, (hW^, X'^,^.y)) for inference of and {U{y),V{y)) = {\\y\\^,X]^y) for inference of ct|. □ 

Thus, by studying the distributions (1.13) and (1.14), we will be able to perform inference after 
selection via the square-root LASSO. To gain insight for the law in (1.13), we first look at the 
simple case where there is no selection. If there were no selection, the above law is simply the law 
of the T-statistic with degrees of freedom n— \E\, 
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where 

»2 _ II ~ -Pe)2/|P _ tvT V \-l vT r> _ V 

CT_E — SLe, — -^E^e- 

n - \h\ 

The selection event is equivalent to {E{y) = E} in the context of this paper, where E is defined 
in (1.6). We can explicitly describe the selection procedure if we further condition on the signs ze, 
that is instead of conditioning on the event {E{y) = E}, we condition on the event {{E{y), %(?/)) = 
{E,ze)} in the laws (1.13) and (1.14). Procedures valid under such laws are also valid under those 
conditioned on {E{y) = E} since we can always marginalize over Therefore, for computational 
reasons we always condition on {[E[y) ^Zsiy)) = (E,ze)}- 

We describe the distributions in (1.13) in detail in Section 3. We will see that they are truncated 
T distributions with the degrees of freedom n — \E\. Based on these laws, we construct exact tests 
for the coefficients Pe- Given that the appropriate laws in the case of a known are truncated 
Gaussian distributions, it is not surprising that the appropriate distributions here are truncated 
T distributions. To construct selective intervals, we suggest a natural Gaussian approximation 
to the truncated T distribution and investigate its performance in a regression problem. Such 
approximation brought much convenience in computation. 

1.4- Organization 

The take-away message of this paper is that selective inference with cr|; unknown is possible in the 
n < p scenario using the square-root LASSO. In Section 2 we describe the square-root LASSO in 
more detail. In particular, we describe the selection events 

[y ■ {E{y),ZEiy)) = iE,ZE)'^ , Ac {I,...,p}, ZB G {-1,1}^. (1.15) 

Following this, in Section 3 we turn our attention to the main inferential tools, the laws (1.13) and 
(1.14), which allow us to perform inference for the coefficients Pe and variance a% in the selected 
model. As an application of the p-values obtained in Section 3, we applied the BHq procedure 
Benjamini & Hochberg (1995) to these p-values and compare the FDR control and power to another 
method designed to control FDR. In Section 4, we also compare the performance of our variance 
estimators with other estimates of the variance. 


2. The Square Root LASSO 


We now use the Karush-Kuhn-Tucker (KKT) conditions to describe the selection event. Recall the 
convex program (1.2) 

Ky) = Pxiy) =argmin||j/-A/3||2-bA- ||/3||i (2.1) 

/3gRp 

as well as our shorthand for the selected variables and signs (1.6), (1.7). 

The KKT conditions characterize the solution as follows: {P{y), z) is the solution and correspond¬ 
ing subgradient of (2.1) if and only if 


X^jy-XPjy)) ^ 
\\y-Xp{y )\\2 
fsign(/3j(y)) iijGEiy) 
[[-1,1] itj^E{y). 


( 2 . 2 ) 

(2.3) 
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We see that our choice of shorthand for corresponds to the E{y) coordinates of the subgradient 
of the 1 1 norm. 

Our first observation, which we had not found in the literature on square-root LASSO, is that 
square-root LASSO and LASSO have equivalent solution paths. In other words, the square-root 
LASSO solution path is a reparametrization of the LASSO solution path. Specifically, 

Lemma 2 . For every (E,ze), on the event {{E{y), Zj^{y)) = {E,ze)} the solutions of the LASSO 
and square-root LASSO are related as 

Ky) = h{y) = h(v){y) (2.4) 


where 


and 


j{y) = X&Eiy) ■ 


f n-\E\ 

{i-\mx^)^zE\\v 


,2,, \\{l-XEXl)y\\l \\{I-PE)y\\l 
^E{y) = --= „_|^| 

is the usual ordinary least squares estimate of in the model M^^e- 

Proof. On the event in question, we can rewrite the KKT conditions using the fact XjS 


(2.5) 


( 2 . 6 ) 


XePe as 


^siy ~ ^E^siy)) — csiy) ■ X ■ ze (2.7) 

X'^^{y — XE^Eiv)) = CE{y) ■ X - Z-E (2.8) 

S\gTL 0 E{y)) = ZE, \\z-e\\oo < 1 (2.9) 


with 

csiy) = \\y - XE^E{y)h- 

Comparing (2.7) with the KKT conditions of LASSO in Lee et al. (2013), this indicates f{y) = 
XcE{y)- We deduce from (2.7) that the active part of the coefficients 

pE{y) = iX^XE)-\x]^y - X ■ CE{y) ■ ze). (2.10) 

Plugging the estimate $E{y), 

y - XE^E^y) = {I - PE)y + A • CE{y) ■ {X'^)^ze. (2-11) 


Computing the squared Euclidean norm of both sides yields 


4(2/) 


\\ii-PE)yr2 

l-A2||(A|)tz^||l 
-2 ( \ n-\E\ 

i-xmx^)^ZE\\r 


□ 
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2.1. A first example 

Before we move on to the general case, it is helpful to look at the characterization of the selection 
event in the case of an orthogonal design matrix. When the design matrix X G has orthogonal 

columns, Pe and ce can be simplified as 


Pe = — XceZe^ ce = ^e 


' n-\E\ 
1-X^\E\' 


The selection event 

I?/ : sign(/3s(j/)) = , 

is decoupled into \E\ constraints. For each i G E, 

Zjxfy ^ . I n- \E\ 

^E{y)- \ji-XW 


( 2 . 12 ) 


The left-hand side of (2.12) is closely related to the inference on Pi, and follows a T-distribution 
with n — \E\ degrees of freedom. The constraint (2.12) is a constraint on the usual T-statistic on 
the selection event which implies one should use the truncated T distribution to tests whether or 
not Pi = 0. 


2.2. Characterization of the selection event 


We now describe the selection event for general design matrices, which will be used for deriving the 
laws (1.13) and (1-14). Using Equation (2.10) in Lemma 2, we see that the event 


is equal to the event 
where 


[y ■■ signipEiy)) = ZE^ 


{y ■ ^E{y) ■ ai,E - Zi^E ■ UE,i{y) <0,i g E} 


(2.13) 

(2.14) 


UEhiy) = 






lefxlh 


and 


o^i.E — ^ ■ Zi^E ■ \\ef Xl,\\2 ■ {1 — X'^\\{X'£PzeWI) (X^Xe) ^ze- 


While the expression is a little involved, it is explicit and easily computable given (X'^Xe) ^ ■ 
Let us now consider the inactive inequalities. The event 


{y ■■ P-b||oo < 1 } 


is equal to the event 


X-CE{y) + * ^ 


< 1 , 


i G -E 


(2.15) 


(2.16) 
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For each i G —E these are equivalent the intersection of the inequalities 


where 


< 1 - xnxl)<ZE 

X 1/2 

{I - PE)y 


1-X^\\{XI)Ue\\ 


A2 


U-E{y) = 


(2.17) 


(2.18) 


W-PE)y\W 

To summarize, the selection event : {E{y)^Zj^{y)) = {E,ZE)^ is equivalent to (2.14) and 
(2.17). 


3. The conditional law 

From the characterization of the selection event in the above section we can now derive the laws 
(1.13) and (1.14). First, the following lemma provides some simplification. 

Lemma 3. Conditioning on (E{y), Zp(y)) = (E,ze), the law for inference of{^, ^) is equivalent 
to 

Qe,ze = P [UE{y), ||(.f — PE)y\\'^ I ^E ■ Oli^E — Zi^E ' UE,i < 0,1 G S] . (3.1) 

Moreover, the statistic U-e is ancillary for both P_e G M^^e ctnd Qe,ze- 
Therefore, the laws (1.13) and (1-14) can be simplified to 

P- [UE,j{y) I \\y\\'^,UE,E\j{y),^E ■ ai,E - zi^e ■ UE,i <0,iG E], 


and 

p [^siy) I UE{y),crE ■ cti,E - Zi^E ■ UE,i < 0, / G Fl] . 

respectively. 

Proof. Per Lemma 1, {X'^y,\\y\\'^) are sufficient statistics for (^,^). Moreover, note Usiy) is 
linear transformation of X"^y and 

bf = \\PEy\\^ + !!(/- PE)y\\\ Pe = Xe{xIXe)-^xI, 

thus it suffices to consider the law of {UE{y),\\{I — F’e) 2/|P) conditioning on {{E{y), zp.{y)) = 
{E,ze)}. 

Moreover, the selection event can be described in two sets of constraints, the ones involving 
Usiy) in (2.14) and the ones involving U-E{y) in (2.17). Notice that U-E{y) is independent of 
(UEiy), d''^iy)), thus we can drop it in the conditional distribution and get (3.1). 

We note that U-E{y) is ancillary for P^;. From the above, the density of Qb.ze is that of P^ 
times an indicator function that does not involve U-E{y). Thus U-E[y) is ancillary for Qe,ze ^-s 
well. 

The simplification of the joint laws leads to that of the marginal laws and the second statement 
holds. □ 
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3.1. Inference under quasi-affine constraints 


The general form of the law Qe,ze (3-1) is that of a multivariate Gaussian and an independent 
of degrees of freedom n— \E\ and satisfying some constraints. These constraints are affine in the 
Gaussian fixing the but not affine in the data. To study the distributions under such constraints, 
we propose the following framework for these quasi-affine constraints. 

Specifically, we want to study the distribution of y ~ '^^I) subject to quasi-affine constraints 


Cy < ap{y) ■ b 


with 


^p{y) = 


\\ii-P)y\\l 

Tr(/ - P) 


where P is a projection matrix, C G b G and 


CP = P, Pfi = y. 


(3.2) 


(3.3) 


Assumptions (3.3) are made to simplify notation. Inference for quasi-affine constraints without 
these assumptions can be deduced similarly. In the example of inference after square-root Lasso, 
assumptions (3.3) are satisfied when we select the correct model E. We denote the above laws as 
Mc,b,p, that is 

P(y G A\Cy < dp{y) ■ b) = M(c,b,p)(A), y - iV(y, a^I). 

Our goal is exact inference for for the directional vector q satisfying Pq = q. Without loss 
of generality we assume ||y||| = I. To test the null hypothesis Hq : q'^q, = 0, we parametrize the 
data into q'^y, the projection onto the direction q, and the orthogonal direction {P — qq^)y. From 
the assumptions (3.3), with some algebra, we can see that is a sufficient statistic for with 
{{P — qq'^)y, ||y|p) being a sufficient statistic for the nuisance parameters. In the following, we will 
prove the law 

q'^y-0 \ {P- qq'^)y, \\y - Oq\\l y - M(c-,6,p). (3.4) 

is a truncated T with degrees of freedom Tr(/ — P) and an explicitly computable truncation set. 
For some set 11 C M let T^|n denote the distribution function of the law of T^\Ti, G : 

T,in(t) = P(TE<tlT,Gfi). (3.5) 

Theorem 4 (Truncated t). Suppose that y ~ The law 

q'^y-0 \ {P- qq'^){y - Oq), \\y - OqWl = TprC-P/n (3-6) 

where 

n = Il{C, 6, P, II (I - P)y\\l + {ri^y - Of, (P - qq^)y, 9). (3.7) 

The precise form of is given in (3.8) below. 

Proof. We use the short hand for the sufficient statistics 

(Pe,G,IFe)(y) = (y'^y - 0, (P - ryry^jy, ||(/- {P - q'^q)){y - 9q)\\l). 
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Since 

Wsiy) = \\y - Oyf - ||(P - vv'^)yf, 


conditioning on {V, \\y — OyW^) is equivalent to conditioning on {V, Wq). 

Our main strategy is to construct a test statistic independent of {V, Wg), this can be easily done 
through the usual T-statistic, 


Tsiy) 


rfy — 9 
^p{y) 


Let d = Tr(/ — P), so 


Wgiy) = I|(/ - P)y\\l + {rfy -Of = d- ap{yf + {rj^y - Of. 


Note that rg is independent of Wg and V. 

We next rewrite the quasi-afhne inequalities (3.2) as 

Ue(y)p + ^ < (^p(y)b 


with 

p = Cr] 

^ = aViy)) = C{ey + Viy)). 

Multiplying both sides by ; we have 

re{y)Wg{y)^^^p + ^{V{y)) ■ ^Jd + r'^iy) < Wg{yf/%. 

This is the constraint on the T-distribution. 

Because of the independence between Tg{y) and {V{y), Wg{y)), its distribution is simply a trun¬ 
cated T-distribution, where 

0(0, 6, P,w,v,9) = S K : t^/w ■ Vi -\- ^i{v) ■ \/d + P < \/w ■ 6| (3-8) 

l<2<nrow(y4) 

and d,^{v) are as above. Each individual inequality can be solved explicitly, with each one yielding at 
most 2 intervals. In practice, we have observed the intersection of the above is not too complex. □ 

Remark 5. Following Lemma 3, we conduct selective inference with the quasi-affine constraints of 
square-root Lasso, where 

C = - diag{zE)diag{{X'^XE)~^)xl, P = Pe, b =-a. 

To test hypothesis Hq : fdj^E = 0; we take rj = CjX"^^. 

3.2. Inference for cr and debiasing under 

The law M(c {, p) is parametric, and in the context of model selection we have observed y inside 
the set 

Cy < dp{y) ■ b. 
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The usual OLS estimates are biased under M(^c,b,P) = In this parametric 

setting, there is a natural procedure to attempt to debias these estimators. 

If we fix the sufficient statistics to be T{y) = {Py, Hj/Hi)) then the natural parameters of the laws 
in are (/r/cr^, — (2cr^)“^). Solving the score equations 


J T{z)M(^c,b,p)■,(^l,a^)idz)-T{y) =0 (3.9) 

for (T^(?/)) corresponds to selective maximum likelihood estimation under M(C'In the 

orthogonal design and known variance setting, this problem was considered by Reid et al. (2013). 
In our current setting, this requires sampling from the constraint set, which is generally non-convex. 

Instead, we consider estimation of each parameter separately based on a form of pseudo-likelihood. 
Unfortunately, in the unknown variance setting, this approach yields estimates either for coordinates 
of or rather than coordinates of /r itself. We propose estimating using pseudo-likelihood 
and plugging in this value to a quantity analogous to M(c ;,,p) but with known variance, i.e. the 
law of ?/ ~ N{y,a'^I),Py = /i with known subject to an affine constraint. This approximation 
is discussed in Section 4.2 below. 

The pseudo-likelihood is based on the law of one sufficient statistic conditional on the other 
sufficient statistics. Therefore, to estimate we consider the likelihood based on the law 


\\{I - P)y\\l\Py, y--M(c. 6 ,p);(;i,a 2 ). (3.10) 

This law depends only on cr^ and can be used for exact inference about though for the parameter 
cr^ an estimate is perhaps more useful than selective tests or selective confidence intervals. 

Direct inspection of the inequalities yield that this law is equivalent to • X^(i-p) truncated 
to the interval [L{Py), U{Py)] where 


L{Py) = max 

^ i-bi>o hi 

TT(P \ ■ 

i\bi<0 hi 


(3.11) 


For fl C K, let q denote the law cr^ • xl truncated to D 

G.,.2^n{t)=F{xl<t\xl&n/a^). 
The pseudo-likelihood estimate i7pL(y) for is the root of 

cr HT,(^j_p){L{Py),U{Py),a'^) - al{y) 


(3.12) 


where 

^ J [0,oo) 

This is easily solved by sampling from cr^x^(/_p) truncated to [L{Py),U{Py)]. 

This procedure is illustrated in Figure la. Note that for observed values of (Tp(y) near the 
truncation boundary the estimate varies quickly with (Tp(y) due to the plateau at the upper limit. 
We remedy this in two simple steps. First, we use a regularized estimate of a under this pseudo¬ 
likelihood. Next, we apply a simple bias correction to this regularized estimate so that when [L, U] = 
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(a) Pseudo-likelihood estimator (b) Regularized pseudo-likelihood estimator 

Fig 1: Estimation of based on the pseudo-likelihood for tr^ under for an interval with 

L = 0. The observed value is d-p{y) = 36 on 100 degrees of freedom trnncated to [0,40]. 


[0, oo) we recover the usual OLS estimator. Specifically, for some 6 we obtain a new estimator as 
the root of 

a H> H„{L, U,a^) + 6 ■ - {I + 9)(fp{y) 

We call this regularized pseudo-likelihood estimate o-p^Miv)- fo practice, we have set 6 = 
so that this regularization becomes negligible as the degrees of freedom grows. The regularized 
estimate can be thought of as the MAP from an improper prior on the natural parameter for . 
In this case, if J = l/(2cr^) is the natural parameter the prior has density proportional to <5'^ ®. 
As H^{0, oo, cr^) = cr^, it is clear that in the untruncated case we recover the usual OLS estimator 
^p(y)- 

4. Applications 

In this section, we discuss several applications of the inferential tools introduced above. In Sec¬ 
tion 4.1, we compare the performance of the variance estimator introduced in Section 3.2 with 
some existing methods. In Section 4.2, we introduce a Gaussian approximation to the truncated 
T-distribution, which is more computation-friendly. The approximation is validated through simu¬ 
lations where the coverages of confidence intervals are close to the nominal levels. Finally, in Section 
4.4, we apply a BHq procedure to the p-values acquired through the inference above. Although we 
do not seek to establish any theoretical results, simulations show that a simple BHq procedure 
applied to the p-values controls the false discovery rate at the desired level and has comparable 
power with existing methods, even in the high-dimensional setting. 


4-1. Comparison of estimators 

As the selection event for the square-root LASSO yields a law of the form we can study 

the accuracy of the estimator by comparing it to other estimators for a in the LASSO literature. 
We compare our estimator, dpp p, to the following: 
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(a) Screening (b) Non-screening 

Fig 2: Comparisons of different estimators for cr^. 


• OLS estimator in the selected model, where a = E is the active set. 

• Scaled LASSO in Sun & Zhang (2011). 

• Minimum cross-validation estimator based upon the residual sum of squares of Lasso solution 
with A selected by cross validation Reid et al. (2013). 

To illustrate the advantage of our method, we consider the high-dimensional setting. The design 
matrices were 1000 x 2000 generated from an equicorrelated Gaussian with correlation 0.3, columns 
normalized to have length 1. The sparsity was set to 40 non-zero coefficients each with a signal-to- 
noise ratio 7 but with a random sign. The noise level cr = 3 is considered unknown. The parameter 
K in (1.5) was set to 0.8. With these settings, the square-root LASSO “screened”, or discovered a 
superset of the 20 non-zero coefficients with a success rate of approximately 30%. Since we do not 
screen most of the time, the model may be sometimes misspecified. But the variance estimator apL 
is consistent with the model specific variance ap- 

Specihcally, the performance of the estimators was evaluated by considering the ratio 'S^{y)/^% 
where a\ is the usual estimate of using the selected variables E, evaluated on an independent 
copy of data drawn from the same distribution. This is the variance one would expect to see in long 
run sampling if fitting an OLS model with variables E under the true data generating distribution. 

We see from Figure 2a that our estimator is approximately unbiased when we correctly recovered 
all variables, outperforming the OLS estimator and the scaled LASSO estimator. Its performance 
is comparable with that of Minimum CV, but with fewer outliers. In the case of partial recovery 
(non-screening), we see that the estimator is quite close to the estimator where part of the 
“noise” in our selected model comes from the bias in the estimator. Our estimator beats all the 
other estimators in this case. Particularly, the Selected OLS estimator consistently underestimates 
the variance, which might lead to inflated test scores and more false discoveries. On the other hand. 
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Minimum CV seeks to estimate instead of which results in large downward bias. 

In fact, when (iji,Xi) are independent draws from a fixed Gaussian distribution, then for any 
E, the model Mu^e is correctly specified in the sense that the law of y\XE belongs to the family 
M^ e- In this setting the quantity is an asymptotically correct estimator trl; = X&T:{yi\xi^E)- 
Hence, we see that the pseudo-likelihood estimator may be considered a reasonable estimator when 
the square-root LASSO does not actually screen. 

4-2. A Gaussian approximation to 

In this section, we introduce a Gaussian approximation to the truncated T-distribution which we 
use in computation. 

The selective distribution Qe,ze derived from some Pp S is used for inference about the 

parameters Pe- Let rj be the normalized linear functional for testing Hq : jSj^E = O-, then for any 
value of 0, this distribution is restricted to the sphere of radius \\y — 0 Xj \\2 intersect the affine 
space {z : X'^^^z = Gall this set S{\\PE\j{y — 9ri)\\2, X'^^^y) The restriction of Qe,ze I® 

S{\\PE\j{y — 6 r])\\ 2 , X]^s^-y) is the law Pp restricted to S{\\PE\j{y — Orf), X'^^^-y) intersected with 
the selection event. For \E\ not large relative to n by the classical Poincare’s limit Diaconis & 
Freedman (1987), the law of rfy — 9 under Pp restricted to S{\\PE\j{y — 9 r])\\ 2 , X'^s^-y) that is 
close to a Gaussian with variance \\PE\j{y — 9 ri)\\ 2 /{n — \E\ -|- 1). Thus, we might approximate its 
distribution under Qe,ze by ^ truncated Gaussian. Furthermore, we can approximate the variance 
by dp{y)‘^. We summarize the Gaussian approximation in the following remark. 

Remark 6 (Approximate distribution). Suppose we are interested in testing the hypothesis PIq ■ 
rf"p = 9 in the family for some rj G row{C). We propose using the distribution 

C{ 7 fz -9\{P - 7777 ^) 2 , Cz < ap{y)b), z - N{p, a^iy)I). 

We condition on {P — rjrj^)z as we have assumed Pp = p in defining Wi.(^Q b,p) is a sufficient 

statistic for the unknown parameter {P — rirj^)p. 

To validate the approximation, we use a similar simulation scenario to the one in Section 7 
of Fithian et al. (2014). We generate rows of the design matrix A 150 X 200 from an equicorrelated 
multivariate Gaussian distribution with pairwise correlation p = 0.3 between the variables. The 
columns are normalized to have length 1. The sparsity level is 10, with each non-zero coefficient 
having value 6. Results are shown in Table 1. 


Level 

Coverage 

0.850 

0.860 

0.900 

0.905 

0.950 

0.947 

0.970 

0.968 


Table 1 

Coverage of confidence intervals using Gaussian approximation based on forming 10000 intervals. 
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4-3. Regression diagnostics 


Recall the scaled residual vector U-E{y) in (2-18) is ancillary under the laws Pg and Qe,ze- U-E{y) 
follows a uniform distribution on the n-dimensional unit sphere intersecting the subspace determined 
by / — Pe, truncated by the observed constraints (2.17). As U-E{y) is ancillary, we can sample 
from its distribution to carry out any regression diagnostics or goodness of fit tests. For a specific 
example, we might consider the observed maximum of the residuals ||t/_£;(?/)||oo- 

Another natural regression diagnostic might be to test whether individual or groups of variables 
not selected improve the fit. Specifically, suppose G is a subset of variables disjoint from E. Then, 
the usual F statistic for including these variables in the model is measurable with respect to U-e'- 


PG\GiJE{y) 


\\{Pg^e - PE)y\\ll\G\ 
\\{I-PGuE)y\\l/{n-\GUE\) 
\\{PGVE-PE)U-E{y)\\l/\G\ 
\\{I-PGuE)U-E{y)\\l/{n-\GyjE\y 


Therefore, a selectively valid test of 

Ho ■ Pg\gue = 0 

can be constructed by sampling U-e under its null distribution and comparing the observed F 
statistic to this reference distribution. Details of these diagnostics are a potential area of further 
work. 


4-4- Applications to FDR control 

Based on the truncated-t distribution derived in Theorem 4, it is easy to construct tests that 
control the selective Type I error (1.9). In this section, we attempt to use our p-values for multiple 
hypothesis testing purposes. We apply the BHq procedure (Benjamini & Hochberg 1995) to the 
p-values and compare with the procedure proposed by Barber & Candes (2016). To ensure the 
fairness of the comparison, we generate the data according to the data generation mechanism in 
Section 5 of Barber & Candes (2016), where n = 2000, p = 2500, k = 30, and X is generated as 
random Gaussian design with correlation p. In the simulations, we vary k in (1.5) and also include 
the choice of A according to the 1 standard error rule for reference. The comparison is in Figure 3 
below. We see that first our procedure is relatively robust to the choice of k in both FDR control and 
power across different correlations p. Secondly, for all choices of k, our procedure controls the FDR 
at 0.2 across all the correlation coefficients p. Moreover, it enjoys an approximately 20% increase 
in power compared with Figure 1 in Barber & Candes (2016). 
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Fig 3: FDR control and power for square-root Lasso across different correlations p. 
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